Skip to content

[SPARK-10330] Add Scalastyle rule to require use of SparkHadoopUtil JobContext methods#8521

Closed
JoshRosen wants to merge 3 commits into
apache:masterfrom
JoshRosen:SPARK-10330-part2
Closed

[SPARK-10330] Add Scalastyle rule to require use of SparkHadoopUtil JobContext methods#8521
JoshRosen wants to merge 3 commits into
apache:masterfrom
JoshRosen:SPARK-10330-part2

Conversation

@JoshRosen

@JoshRosen JoshRosen commented Aug 29, 2015

Copy link
Copy Markdown
Contributor

This is a followup to #8499 which adds a Scalastyle rule to mandate the use of SparkHadoopUtil's JobContext accessor methods and fixes the existing violations.

@SparkQA

SparkQA commented Aug 29, 2015

Copy link
Copy Markdown

Test build #41788 has finished for PR 8521 at commit 58dd847.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class JavaTrainValidationSplitExample

@JoshRosen

JoshRosen commented Aug 30, 2015

Copy link
Copy Markdown
Contributor Author

Jenkins, retest this please.

@JoshRosen

JoshRosen commented Aug 30, 2015

Copy link
Copy Markdown
Contributor Author

(Re-testing pre-emptively in case of flakiness)

@SparkQA

SparkQA commented Aug 30, 2015

Copy link
Copy Markdown

Test build #41789 has finished for PR 8521 at commit d9cb6df.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 30, 2015

Copy link
Copy Markdown

Test build #41792 has finished for PR 8521 at commit d9cb6df.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class JavaTrainValidationSplitExample
    • class KMeans @Since("1.5.0") (

zsxwing
zsxwing reviewed Aug 30, 2015
View reviewed changes
Comment thread core/src/main/scala/org/apache/spark/SparkContext.scala

@zsxwing zsxwing Aug 30, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JoshRosen I didn't change job.getConfiguration last time because they uses org.apache.hadoop.mapreduce.Job whose getConfiguration is compatible. However, +1 for this change since it is good for maintenance.

@JoshRosen

JoshRosen commented Sep 9, 2015

Copy link
Copy Markdown
Contributor Author

Jenkins, retest this please.

@SparkQA

SparkQA commented Sep 9, 2015

Copy link
Copy Markdown

Test build #42179 has finished for PR 8521 at commit d9cb6df.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@JoshRosen

JoshRosen commented Sep 9, 2015

Copy link
Copy Markdown
Contributor Author

Since this still passes tests, can we merge this now?

@JoshRosen

JoshRosen commented Sep 10, 2015

Copy link
Copy Markdown
Contributor Author

Jenkins, retest this please.

@JoshRosen

JoshRosen commented Sep 10, 2015

Copy link
Copy Markdown
Contributor Author

Going to retest and will merge once tests pass again.

@SparkQA

SparkQA commented Sep 11, 2015

Copy link
Copy Markdown

Test build #42296 has finished for PR 8521 at commit d9cb6df.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus

marmbrus commented Sep 11, 2015

Copy link
Copy Markdown
Contributor

LGTM

@JoshRosen

JoshRosen commented Sep 12, 2015

Copy link
Copy Markdown
Contributor Author

Jenkins, retest this please.

@JoshRosen

JoshRosen commented Sep 12, 2015

Copy link
Copy Markdown
Contributor Author

Going to merge this into master.

@asfgit asfgit closed this in b3a7480 Sep 12, 2015
@JoshRosen JoshRosen deleted the SPARK-10330-part2 branch September 12, 2015 23:26
@SparkQA

SparkQA commented Sep 13, 2015

Copy link
Copy Markdown

Test build #42377 has finished for PR 8521 at commit d9cb6df.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@JoshRosen @SparkQA @marmbrus @zsxwing